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^ (54) Title: MULTIPLE TONE DATA ENCODING AND DECODING IN VIDEO CARRIER SIGNAI-S 

(57) Abstract: The invention describes a system and method for encoding and decoding data from modulated carrier signals received 
2J optically from a raster-scanned display. Data is imposed upon the luminance portion of a TV picture. In an embodiment of the 
^ invention, the data is superimposed at a rate four times per field and at frequencies approximately 25 kHz for a mark and 25.6 kHz 
^ for a space. Other embodiments involve different frequencies for mark and space, alternatively, bit and bit bar. In one embodiment, 
^ the data is imposed at four frequencies, the foiu frequencies corresponding to the signals {(X) 01 10 1 1 }. In another embodiment, 
S the data is imposed using ei^t frequencies, conresponding to the signals {000 001 010 01 1 100 101 1 10 1 1 1 ). Thus the invention 

multiplies the bandwidth of conventional carrier-signal based encoding systems. The decoder may include a photo receiver, and 
^5 amplifier, and a filter circuit coupled to a DSP processor for decoding the signals received from a television. The DSP processor 
^ employs Discrete Fourier Transforms, or DFTs, to decode the signal. This decoding technique allows signals to be discerned more 
^ robustly than permissible in conventional systems. 
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MULTIPLE TONE DATA ENCODING AND DECODING IN VIDEO 

CARRIER SIGNALS 

BACKGROUND OF THE INVENTION 

5 Field of the Invention 

This invention relates to the field of transmission of non- visual data via 
a video signal. In partictdar, the invention relates to techniques for encoding and 
transmitting digital data in a visible optical signal. 
Description of the Related Art 

1 0 The present invention provides for transmission of digital data along 

with any standard television or computer video signal for reception by a remote 
device held near the television screen. 

Methods of transmission of data together with television signals from a 
remote video broadcast site to home television viewers over conventional 

1 5 broadcast channels include, but are not limited to: Vertical Blanking Interval 
(VBI) data transmissions— typically Teletext and closed captioning for the 
hearing impaired; FM broadcast radio subcarrier, and Cellular telephone-like 
data transmission systems. Only television sets specially equipped with 
decoders for displaying video text in synchronization with the television's raster 

20 scan electronics are able to display the captions making such systems complex 
and not readily available to most television viewers. 

Subliminal encoding of data has been used to minimize adverse impact 
on the effective viewing area of the video program material. A method for 
subliminally encoding binary data, within the vievring area of a video program 

25 image that is substantially invisible to a viewer of the television is described in 
U.S. Pat. No. 4,807,03 1 issued to Broughton et al. on Feb. 21, 1989. According 
to the invention of Broughton et al., the data encoding method involves 
modulating a video signal at frequencies that are related to multiples and sub- 
multiples of the horizontal line rate, to produce a subtie video subcanier such as 
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pulse modulation (PM), phase modulation (PM), amplitude modulation (AM), 
frequency modulation (FM), time or pulse interved modulation (PIM), frequency 
shift keying (FSK), retum-to-zero (RZ), non-return-to-zero (NRZ), or any other 
of a variety of spatio-temporal modulation and coding techniques. Although the 
5 modulation is within the viewing area of the television screen, it remains 

substantially invisible to the viewer because of its relatively low intensity. The 
encoded information can be recovered in a remote receiver aimed at the 
displayed television image to capture the information. 

The problems associated with the Broughton arrangement are caused by 

10 generating the subliminal encoding by selectively increasing the intensity of one 
of each pair of adjacent scan lines in each frame of the video image. The 
problem with this approach is that the encoded information causes changes in 
the frequency domain representation of the video portion of the television 
signal, which occur at one-half of the line rate. Since there is a strong signal 

1 5 present at this frequency even without data present, the encoded information is 
difficult to capture in the receiver. 

An improvement over Broughton et al. is described in U.S. Patent No. 
5,633,766 issued to Sizer, II, on Sep. 2, 1997 where the digital information is 
encoded by modulating a carrier signal or tone, using, for example, amplitude 

20 shift keying (ASK) or frequency shift keying (FSK), and the modulated carrier 
is then added to the video signal selectively, only in portions of the television 
program that (a) are not likely to be perceptible by a viewer, and (b) are of 
sufficient intensity to transmit the data. However, the subliminal nature of the 
luminance modulation necessitates transmission of a relatively low intensity 

25 signal. 

Broughton et al. and Sis^r, II, both involve transmitting data in a manner 
which is imperceptible to the viewer in order to avoid complaints from non- 
subscribing consumers and FCC restrictions. However, both of these techniques 
suffer from limitations in performance and bandwidth. In particular, the system 
30 used in Sizer uses two tones for encoding a bit and a bit bar signal. This system 
is not extensible to multiple tones, however, as the decoding techniques 
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suggested in Sizer are unable to distinguish between multiple frequencies 
spaced closely together. 

Accordingly, it is an object of the present invention to provide a method 
for encoding and digital data in a visible optical signal and transmitting the data 
5 during a remote video broadcast to a remote device by use of a carrier signal 
superimposed with two or more tones. Another object of the invention is to 
provide a robust decoding system, which can reliably distinguish between 
multiple frequencies spaced closely together. 

10 SUMMARY OF THE INVENTION 

The invention comprises a system and method for encoding and 
decoding data from modulated carrier signals received optically from a raster- 
scanned display. Data may be imposed upon the limiinance and/or chrominance 
portion of the display. In an embodiment of the invention, the data is 

1 5 superimposed at a rate four times per field and at frequencies approximately 25 
kHz for a mark and 25.6 kHz for a space. Other embodiments involve different 
frequencies for mark and space, alternatively, bit and bit bar. In one 
embodiment, the data is imposed at four frequencies, the four frequencies 
corresponding to the signals {00 01 10 11}. In another embodiment, the data is 

20 imposed using eight frequencies, corresponding to the signals {000 001 010 01 1 
1 00 101 110111}. Thus the invention multiplies the bandwidth of conventional 
carrier-signal based encoding systems. 

A decoder in the invention may include a photo receiver, an amplifier, 
and a filter circuit coupled to a DSP processor for decoding the signals received 

25 from the display. The DSP processor employs Discrete Fourier Transforms, or 
DFTs, to decode the signal. This decoding technique allows signals to be 
discerned more robustly than permissible in conventional systems. Execution 
of the DFT may include use of a Fast Fourier Transformation, or FFT. 
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Embodiments of the invention include a method for decoding an optical 
signal from a raster-scanned display, wherein the optical signal is received in a 
decoder imit from the raster scanned display. The optical signal in such 
embodiments may include a carrier signal modulated with a plurality of 
5 frequency keys. 

The optical signal is processed in a digital signal processor by 
transforming a plurality of samples of the optical signal to the plurality of keys 
by a Discrete Fourier Transform. In some embodiments, the plurality of 
frequency keys comprises four or more frequency keys. In some such 

10 embodiments, each of the four or more frequency keys is mapped to a 2-tuple. 
In other embodiments, the plurality of frequency keys comprises eight or more 
frequency keys. In some such embodiments, each of the plurality of frequency 
keys are mapped to a 3-tuple. 

The carrier signal may be luminance or chrominance modulated. In 

1 5 some embodiments, a Fast Fourier Transform is to the plurality of samples to 
perform the DFT. In some embodiments, the modulated carrier signal is 
subliminal, thus imperceptible to viewers. The raster scanned display may be a 
television operating on NTSC, PAL, or SECAM. The display may be interlaced 
or progressive scanned. These and other embodiments shall be described in 

20 greater detail infra. 
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BRIEF DESCMPTION OF THE FIGURES 

Fig. 1 illustrates a raster-scanned display and a decoder used in 
embodiments of the invention. 

Fig. 2 is a schematic illustration of a decoder circuit according to an 
5 embodiment of the invention. 

Fig. 3 is an envelope of data encoded in a carrier signal. 

Fig. 4 shows spectral peaks at two frequencies according to 
embodiments of the invention. 

Fig. 5 is an expanded view of spectral plats according to embodiments 
10 of the invention. 

Fig. 6 is a flowchart for application of a DFT to decode samples of the 
optical signal. 
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DETAILED DESCRIPTION 

A. System Overview 

Figure 1 illustrates the operation of an encoding/decoding system as 
used in the invention. Data is imposed upon the luminance portion of a raster 
5 scanned display 100, In alternative embodiments, data may be imposed upon 
the chrominance portion of the display 100. The data is encoded by modulating 
the luminance, or alternative chrominance, of a carrier signal fed to the raster 
scanned display 100. In embodiments of the invention, the modulated carrier 
signal is subliminal, i.e., imperceptible to a viewer. 

10 Embodiments of the invention employ Frequency Shift Keying, or FSK, 

to encode data signals. Some such embodiments encode a bit signal and a bit 
bar signal with different frequency keys. In an embodiment of the invention, 
the data is superimposed at a rate four times per field and at frequencies 
approximately 25 kHz for a bit and 25.6 kHz for a bit bar. Other embodiments 

1 5 involve different frequencies for bit and bit bar. In one embodiment, the data is 
imposed at four frequencies, the four fi*equencies corresponding to the bit pairs 
{00 01 10 11}. In another embodiment, the data is imposed using eight 
frequencies, corresponding to the bit three-tuples {000 001 010 01 1 100 101 
110 111}. 

20 A decoder 102 may include a photo receiver 200, and amplifier 202, and 

a filter circuit 204, as illustrated schematically in Fig. 2. The decoder also 
includes a DSP processor 206 for decoding the signals received from the 
television 100. The DSP 206 employs Discrete Fourier Transforms, or DFTs, to 
decode the signal. The DSP may include a Field Programmable Gate Array, or 

25 FPGA, for performing the DFT. Examples of FPGAs which may be used in the 
decoder 102 include those developed by Xilinx, Inc., or alternatives apparent to 
those skilled in the art. The decoder 102 may be a portable handheld unit, or 
alternatively, a stationary device. 
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B. Numerics for Frequency Shift Keying 

An embodiment of the invention employs Frequency Shift Keying, or 
FSK, for signaling and demodulation. In one such embodiment, the decoder 
samples the optical signal at a rate of 16.121 856 MHz. Selecting 161 and 157 
5 as divide ratios, an average of 161 yields 1 001 36 Hz, and an average of 157 
yields 102687 Hz. As will be apparent to one skilled in the art, multiplying by 
proper sequences for inphase and quadrature, i.e., I and Q, yields frequencies: 

25.034 kHz and 25.672 kHz. 
These frequencies may be used for bit and bit bar. Alternative frequencies for 
10 bit and bit bar will be apparent to those skilled in the art. 

Embodiments of the invention are directed towards an NTSC broadcast 
standard. In such television systems, there are 525 lines total or 262.5 lines per 
field of which 22 are retrace, for a total of 242.5 useful per field. Thus, to 
encode 4 bits per filed, each bit may last up to 60 lines per bit. In such an 
15 embodiment, each line is of duration l/(30*525)=63.49 [iscc 
which makes a symbol duration of aroimd 

60*63.49=3.8095 msec (262.5 baud). 

Symbol duration times, as well as line, field, and frame rate, will differ for other 
20 television standards, such as PAL and SECAM. It will be clear to one skilled in 
the art that the FSK techniques described for NTSC are easily portable to other 
broadcast standards, including but not limited to PAL and SECAM. 

C. Signal Decoding Using the DSP 

Figure 3 shows an envelope of data 300 imposed upon a carrier signal 
25 sent to the display 100. Figure 4 illustrates an effective demodulation of the 
modulated signal received at the decoder 102 comprising 20 milliseconds of 8 
bit data sampled at a 500 kHz rate. The originating luminance from the display 
100 is the carrier signal modulated with an additive signal at the 25 kHz and 
25.6 kHz FSK frequencies according to an embodiment discussed above. 
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The received signal is gathered in the decoder 102. The demodulation is 
performed by a DFT, implemented with a series of overlapping Fast Fourier 
Transforms, or FFTS, of size 1200 samples apiece (2.4 millisecond window), 
taken with an overlap of 100 samples (0.2 milliseconds overlap). This may be 
5 performed in the DSP 206. Other sample sizes may be also be used, as will be 
apparent to one skilled in the art. For each FFT the frequency of the maximum 
of the spectral peak is determined. An interpolation algorithm executed in the 
DSP 206 provides precise frequency to a resolution of about 4 Hz. Figure 4 
shows examples of such spectral peaks. In Figure 4, the frequencies 25.6 kHz 
10 400 and 25 kHz 402 are clearly discernible and the signal is oscillating between 
the two. 

Figure 5 illustrates the spectral of several sections of the data 
corresponding to primary frequencies of 25 500 and 25.6 kHz 502. As will be 
apparent to one skilled in the art, the respective spectral peaks 500 502 are 
15 clean, well defined and with reasonable excess spreading. 

D. Analysis of DFT on Two*ToneFSK 

A decoding technique used in embodiments of the invention includes 
DFT components at two frequencies by sampling the input signal at 16.121856 
MHz, decimating by 161 and 157, to get effective sample rates at 16.121856 

20 MHz/161 and 16.121856 MHz/157 and then evaluating the DFT at one-fourth 
of the effective sample rate. As will be apparent to one skilled in the art, such 
DFT's are easily done since they are performed by multiplying the resulting data 
by a sequence 1, j, -1, -j, for the duration of a bit period. For a 4 msec symbol 
period and aroimd 100 kHz sample rate the DFT size is approximately 400 

25 samples in length. Other effective sample rates will be apparent to those skilled 
in the art. In an embodiment of the invention, the initial decimation is done by 
simply integrating for 161, or 157 samples and dumping the result. The initial 
integrate and dump may be modeled as passing the original signal through a 
boxcar filter of duration 161 or 157 and then sampling the result at the 

30 decimated rate. 
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E. Decoding Bv Use of Discrete Fourier Transforms 

The decoding technique used by the decoder 102 is illustrated in the 
flow chart of Figure 6. The DFT 206 samples the signal received from the 
display 600. A Discrete Fourier Transform is performed on the samples 602 to 
5 retrieve frequency keys. The DFT is computed by use of a Fast Fourier 
Transformation, or FFT. In embodiments of the invention the DFT 602 and 
FFT 604 are performed in the DSP 206 of the decoder 102. The frequency keys 
retrieved are then mapped to bit sets 606 to decode the signal. The mapping 
step 606 may be illustrated for the example described above, in which the 

10 frequency key 25 KHz is mapped to a bit and the frequency key 25.6 KHz is 
mapped to a bit bar. 

The measurements shown in figures 4 and 5 support the high 
performance of the two level FSK signaling described above. The invention 
can also support foxu- tones and eight tones, thereby doubling and tripling the bit 

15 rates, respectively. 

As a non-limiting example, note that the 2-level FSK described above 
may be expanded to four tones, for example, by using divide ratios 161, 159, 
157 and 155 which yields final frequencies {25034,25349,25672,26003} which 
are spaced by around 300 Hz. For a 4 msec symbol period there is essentially 

20 no correlation between the tones. These four frequencies may be mapped to the 
following bit sets, each of which is a 2-tuple: 



Table 1 



Frequency Keys (in KHz) 


Bit Sets 


25.034 


00 


25.349 


01 


25.672 


10 


26.003 


11 



25 Other frequency keys and mappings to 2-tuple bit sets will be apparent 

to one skilled in the art. Thus, with four tones, the bit rate is doubled. 
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If instead we use divide ratios separate by one, then the separation 
between tones would be around 1 50 Hz, which for a 4 msec symbol period 
would provide correlation between adjacent tones of 0.5. The total span of all 
such tones would be around 1250 Hz. It will be apparent to one skilled in the 
5 art that eight such tones, or frequency keys, may be mapped to eight different 3- 
tuple bit sets. This would provide a tripling of the bit rate. 

F. Conclusion 

The foregoing description of various embodiments of the invention has 
been presented for purposes of illustration and description. It is not intended to 
10 limit the invention to the precise forms disclosed. Many modifications and 
equivalent arrangements will be apparent. 
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CLAIMS 

What is claimed is: 

1 . A method for decoding an optical signal from a raster-scanned display, 
the method comprising: 

5 receiving the optical signal from the raster scanned display, the optical 

signal comprising a modulated ceirrier signal, the modulated carrier 
signal including a plurality of frequency keys, the optical signal 
received through a photodiode on an electronic device; 
processing the optical signal in a digital signal processor, the processing 
10 the optical signal including 

transforming a plurality of samples of the optical signal to the 
plurality of keys by a Discrete Fourier Transform. 

2. The method of claim 1 wherein the plurality of frequency keys 
comprises four or more frequency keys. 

IS 3 . The method of claim! , wherein the plurality of frequency keys 
comprises eight or more frequency keys. 

4. The method of claim 1 , wherein the plurality of frequency keys 
comprises two or more frequency keys. 

5. The method of claim 4, wherein the two or more frequency keys 
20 includes a first frequency key of on or about 25 kHz. 

6. The method of claim 4, wherein the two or more frequency keys 
includes a second frequency key of on or about 25.6 kHz. 

7. The method of claim 1 , wherein the modulated carrier signal includes a 
modulated chrominance level. 

11 
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8. The method of claim 1 » wherein the modulated carrier signal includes a 
modulated luminance level. 

9. The method of claim 1, wherein the transforming the plurality of 
samples further includes applying a Fast Fourier Transform to the plurality of 

5 samples. 

1 0. The method of claim 1 , wherein the modulated carrier signal is 
subliminal. 

1 1 . The method of claim 1 , wherein the raster scanned display is a television 
operating on an NTSC standard. 

10 12. The method of claim 1 , wherein the raster-scanned dispaly is a television 
operating on a SECAM standard. 

1 3 . The method of claim 1 , wherein the raster-scanned display is a 
television operating on a PAL standard. 

14. The method of claim 1, wherein the raster-scanned display is an 
1 5 interlaced display. 

15. The method of claim 1 , wherein the raster-scanned display is a 
progressive scan display. 

16. The method of claim 1 , wherein the raster-scanned display is a computer 
monitor. 

20 17. A method of encoding a data signal in a optical signal displayed on a 
raster scanned display, the optical signal including a carrier signal, the method 
comprising: 

encoding a first plurality of bit sets in the data signal by modulating the 
carrier signal with a first frequency key; 
25 encoding a second plurality of bit sets in the data signal by modulating 

the carrier signal with a second frequency key; 

12 
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encoding a third plurality of bit sets in the data signal with a third 
frequency key by modulating the carrier signal with a third 
frequency key. 

1 8, The method of claim 17, further comprising: 

S encoding a fourth plurality of bit sets in the data signal by 

modulating the carrier signal with a fourth frequency key. 

1 9. The method of claim 1 8, wherein the modulating the carrier signal with 
the first frequency key includes modulating a luminance level of the carrier 
signal with the first frequency key. 

10 20. The method of claim 1 9, wherein the modulating the carrier signal with 
the second frequency key includes modulating the luminance level of the carrier 
signal with the second frequency key. 

21 . The method of claim 20, wherein the modulating the carrier signal with 
the third frequency key includes modulating a luminance level of the carrier 

1 S signal with the third frequency key. 

22. The method of claim 21 , wherein the modulating the carrier signal with 
the fourth frequency key includes modulating the luminance level of the carrier 
signal with the fourth frequency key. 

23. The method of claim 18, wherein the modulating the carrier signal v^th 
20 the first frequency key includes modulating a chrominance level of the carrier 

signal with the first frequency key. 

24. The method of claim 23, wherein the modulating the carrier signal with 
the second frequency key includes modulating a chrominance level of the 
carrier signal with the second frequency key. 

25 25. The method of claim 24, wherein the modulating the carrier signal with 
the third frequency key includes modulating a chrominance level of the carrier 
signal with the third frequency key. 
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26. The method of claim 25, wherein the modulating the carrier signal with 
the fourth frequency key includes modulating a chrominance level of the carrier 
signal with the fourth frequency key. 

27. The method of clzdm 1 8, wherein each bit set in the first plurality of bit 
5 sets corresponds to a first 2-tuple. 

28. The method of claim 27, wherein the first 2-tuple is {00 } . 

29. The method of claim 27, wherein each bit set in the second plurality of 
bit sets corresponds to a second 2-tuple. 

30. The method of claim 29, wherein the second 2-tuple is {01 }. 

10 31. The method of claim 29, wherein each bit set in the third plurality of bit 
sets corresponds to a third 2-tuple. 

32. The method of claim 31, wherein the third 2-tuple is { 10}. 

33. The method of claim 3 1 , wherein each bit set in the fourth plurality of 
bit sets corresponds to a fourth 2-tuple. 

15 34. The method of claim 33, wherein the fourth 2-tuple is { 1 1 }. 

35. An electronic device for decoding data fi:om encoded carrier signals, the 
electronic device comprising: 

a photo detector for receiving the encoded carrier signals optically; 

a bandpass filter coupled to the photodetector, wherein the bandpass 
20 filter filters the encoded carrier signals; 

a digital signal processor coupled to the bandpass filter, the digital signal 
processor including a Discrete Fourier Transformation software for 
decoding the data from the encoded carrier signals. 

36. The electronic device of claim 35, wherein the digital signal processor 
25 includes a Field Programmable Gate Array. 
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37. The electronic device of claim 35, wherein the electronic device is a 
portable hand held unit. 

38. The electronic device of claim 35, wherein the electronic device is 
stationary. 

5 39. The electronic device of claim 35, wherein the encoded carrier signal is 
liiminance modulated. 

40. The electronic device of claim 35, wherein the encoded carrier signal is 
chrominance modulated. 

41 . A method of decoding a data packet encoded in an optical signal 
displayed on a raster scanned monitor, the method comprising: 

receiving the optical signal in an optical receptor; 
sampling the optical signal in a digital signal processor coupled to the 

optical receptor to generate a plurality of samples; 
converting the plurality of samples to a plurality of frequency keys in the 
digital signal processor, the converting the plurality of samples 
including 

applying an algebraic transformation to the samples to generate a 
plurality of frequency keys; 
mapping the plurality of frequency keys to a plurality of bits to decode 
the data packet. 

42. The method of claim 41 , wherein the algebraic transformation is a 
Discrete Fourier Transformation. 

43. The method of claim 42, wherein the applying the algebraic 
transformation further includes applying a Fast Fourier Transformation to the 

25 samples to generate the plurality of frequency keys. 

44. The method of claim 43, wherein the optical signal is subliminal. 
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